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Abstract - Modern NASA planetary exploration missions 
employ complex systems of hardware and software 
managed by large teams of. engineers and scientists in 
order to study remote environments. The most complex 
and successful of these recent projects is the Mars 
Exploration Rover mission. The Computational Sciences 
Division at NASA Ames Research Center delivered a 3D 
visualization program, Viz, to the MER mission that 
provides an immersive, interactive environment for science 
analysis of the remote planetary surface. In addition, 
Ames provided the Athena Science Team with high-quality 
terrain reconstructions generated with the Ames Stereo- 
pipeline. The on-site support team for these software 
systems responded to unanticipated opportunities to 
generate 3D terrain models during the primary MER 
mission. This paper describes Viz, the Stereo-pipeline, 
and the experiences of the on-site team supporting the 
scientists at JPL during the primary MER mission. 

Keywords: Visualization, virtual reality, software 
architectures, simulation, stereo correlation, computer 
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1 Introduction 

The Mars Exploration Rovers (MER) mission is the 
first to have deployed truly mobile robotic science 
platforms on the surface of Mars. The two MER rovers. 
Spirit and Opportunity, have spent over one year 
navigating kilometers of distance and acquiring tens of 
gigabytes of image and spectral data [1]. This represents an 
orders of magnitude increase in mobility and data 
acquisition relative to previous Mars surface missions. To 
handle the complexity of managing these rovers, many 
disparate software tools that could not be fully tested prior 
to operations, were required to function cohesively in a 
mission critical environment. Additionally, diverse groups 
of people, in fields ranging from geology to software 
engineering to network communications, were required to 
work effectively in an organization that encompassed much 
more than any one individual's area of expertise. The 
challenge these circumstances created and the ultimate 
success that the MER mission experienced demonstrate the 
need for, and rewards of, employing a complex “system of 
systems” in modem planetary exploration. 

Mars Exploration is a high priority for NASA. In the 
coming years even more ambitious surface missions, such 


as the Mars Science Laboratory, will be attempted and 
possibly culminate in human exploration of the planet. As 
such, it is important for the participants in the current 
generation of robotic exploration to document and analyze 
their experiences dealing with complex interdisciplinary 
systems in order to better prepare for these future missions. 

In this paper, we describe the visualization and surface 
reconstruction software provided to the MER mission and 
Athena Science Team by the Autonomous Systems and 
Robotics (ASR) area at NASA Ames Research Center 
(ARC), as well as the experiences of the on-site team 
supporting the scientists using this software at the Jet 
Propulsion Laboratory (JPL) during the primary MER 
mission. 

1.1 Background 

The ASR area at NASA ARC was responsible for 
delivering a science operations software element to the 
MER mission - a 3D visualization program named Viz. 
Designed to help scientists understand a wealth of data and 
provide enhanced situational awareness for science 
activities, Viz provided mission scientists with an easy to 
use, immersive, interactive 3D visualization environment. 
In addition to the Viz environment, Ames provided the 
Athena Science Team with high-quality terrain 
reconstructions. This capability was delivered with the 
Ames “Stereo-pipeline,” a software tool that generates 
dense 3D meshes from stereo image pairs. The ARC Viz 
support team assisted Viz users and, in particular, focused 
on responding to unforeseen opportunities to generate 3D 
data products - for example, Microscopic Imager (MI) data 
was used to create finely detailed rock surface models, and 
a mix of orbital and descent imagery was used to create 
digital elevation models of the area surrounding the MER- 
A landing site. 

NASA ARC has been involved in the research and 
development of immersive 3D visualization and surface 
reconstruction techniques since the early 1990s [2]. In 
particular, the ASR (formerly the Autonomy and Robotics 
Area) has focused on applying these technologies in the 
context of planetary exploration missions for science 
understanding and situationaf awareness. In the mid-1990s, 
NASA ARC developed the Ames Stereo-pipeline surface 
reconstruction software and the “MarsMap” visualization 
environment and provided these tools as a technology 



demonstration to the Mars Pathfinder (MPF) mission. The 
Ames Stereo-pipeline implemented a 3D-from-stereo 
capability for automatically producing accurate, high- 
resolution, texture-mapped, 3D terrain models at downlink 
data rates. MaxsMap could display the terrain models 
generated by the Stereo-pipeline in an immersive, 
interactive environment and provided science measurement 
and analysis tools [3]. 

The value of such immersive visualization aids for 
situational awareness was quickly demonstrated as 
scientists planned operations for MPF’s Sojourner rover. 
Navigation hazards that were not visible in the 2D imagery 
or would have required manual photogrammetry to 
determine, were readily apparent in the MarsMap 
environment. In addition, MarsMap’s measurement tools 
and annotation capabilities helped scientists determine 
distributions of rock sizes and wind streak directions 
- clues to the processes at work in an area. 

Although quite effective, MarsMap was developed 
specifically for the MPF mission and was not architected 
for adaptability to other missions or applications. 
Leveraging the experience with MarsMap, ARC developed 
Viz for release to the Mars Polar Lander (MPL) mission as 
a next generation visualization environment with 
adaptability and extensibility as key architectural 
considerations. The concept of a “visualization server” was 
introduced, separating core 3D visualization functionality 
from application or mission specific capabilities and user 
interface (UI) components. 

Also for MPL, a faster more robust version of the 
Ames Stereo-pipeline correlator was developed and 
integrated with JPL image rectification and camera 
calibration software. The integrated surface reconstruction 
software (the “James” pipeline) formed part of the 
mission’s Ground Data Systems (GDS) downlink 
processing pipeline. Unfortunately, the mission was cut 
short due to a spacecraft fault during landing, and as a 
result Viz and the James pipeline only saw use during the 
mission in pre-landing operational readiness tests. 

For the MER mission following MPL, Viz was 
provided as a Class B (mission enhancement) science tool. 
Although the Ames Stereo-pipeline only saw incremental 
changes, the Viz user interface was re-implemented for 
improved mobility support. In addition, the Viz 
architecture was extended to provide better support for a 
seamless user experience, additional simulation tools were 
integrated, and cross-platform portability was improved. 
Viz and the Ames Stereo-pipeline are described in some 
detail in the following section as they were deployed for 
MER. 

In addition to planetary exploration missions, Viz and 
the Ames Stereo-pipeline have been used in several “FIDO” 
and “K9” rover field tests between 1999 and 2004 [4][5], 
Viz and the Ames Stereo-pipeline are also core components 
of ARC’s Single Cycle Instrument Placement (SCIP) 
technology development program [6]. 


2 Viz 

Viz is a cross-platform, modular, extensible, 
visualization software package that allows users to 
interactively manipulate and investigate simulated 3D 
environments. The Viz environment was designed to 
support simple quantitative and qualitative scientific 
analysis of the remote planetary surface, as well as 
conceptual planning of science operations. Viz supports 
hardware stereo viewing capabilities utilizing LCD shutter 
glasses, and provides a variety of surface interrogation 
tools, lighting and rover simulation capabilities, and visual 
feedback cues to enhance the scientists understanding of 
surface topography. A web interface, the "Viz Browser," 
provides the Athena Science Team with easy access to Viz 
and available 3D model data. Viz implements a network 
based communication layer providing an inter-process 
component interface, allowing integration of application 
specific tools that would be difficult to implement at the 
binary level due to differing implementation languages or 
incompatible architectures. In addition, the network 
interface provides an infrastructure for distributed 
applications. For MER, extensibility was further enhanced 
through the addition of a binary plug-in interface. This 
provides improved efficiency for inter-component 
communication and better support for tight user-transparent 
integration of components. 

The separation between visualization server and 
application UI inherent in the foundational Viz architecture 
was leveraged for MER development, and the MPL UI was 
completely replaced with a MER relevant UI (see Figure 1) 
that has support for mobility: multiple sites and reference 
frames, multiple 3D view windows, and rover simulation. 




Figure 1. Top: the MPL Viz user interface. Bottom: the 
MER Viz user interface. 

In addition to user interface changes, file input/output 
functionality had to be adapted to handle MER data 
products from the Athena payload and the MER platform. 
As a primary requirement, 3D model file formats defined 
by JPL GDS teams had to be supported in addition to the 
VRML file format generated by the Ames’ Stereo-pipeline. 
T h is work included adaptation to both geometry and image 
file format specifications. Input of annotations (markers for 
science targets) from JPL’s Science Activity Planner (SAP) 
GDS tool was also implemented, and an articulated 3D 
model of the MER rover was adapted for use within the 
Viz environment. 

NASA ARC provided a number of software tools and 
capabilities to the MER mission, and the Viz team worked 
closely with other ARC teams to leverage synergies where 
possible and coordinate support. Viz plug-ins were 
developed to obtain data from ARC’S Collaborative 
Information Portal (CIP) database infrastructure, and to 
allow Viz to work as a client within ARC’S “MERBoard” 


electronic whiteboard environment. We also worked closely 
with the ARC Microscopic Imager (MI)' Toolkit team 
during the mission support phase to provide scientists with 
a single access point for data products generated by the two 
teams. 

As delivered to the MER mission. Viz implemented 
the following functionality: 

• Interactive 3D fly- through 

• Stereo view (hardware and anaglyph) 

• Interactive time-of-day analysis: shadow simulation, 
sun and planetary body vectors 

8 Science measurement: location, distance, surface area, 
etc. 

• Situational awareness: rover pose and viewpoint 
simulation 

• Site understanding: terrain elevation profiles, color 
coded elevation and slope maps 

• Rendering effects (fog, backgrounds, etc.) 

• Compatibility with JPL generated GDS data products 

• Display of SAP targets 

• Compatibility with ARC’S CIP and MERBoard 

From a user’s standpoint the primary new Viz 
functionalities implemented for MER were: multiple site 
support, multiple 3D views, rover simulation, and shadow 
simulation. 

2.1 Software Architecture 

Viz is implemented in the C++ programming 
language. All toolkits or libraries used were either 
developed at ARC, have freely available source code, or are 
Open Source. The two primary toolkits used in Viz are 
Openlnventor from SGI [7], for basic 3D scene graph 
functionality and management, and Trolltech's Qt cross- 
platform Open Source application framework [8], 
Openlnventor was originally developed commercially by 
SGI, but was open-sourced in 2001. 

Although originally selected as a UI toolkit, Qt has 
broad functionality and is now used throughout Viz. The 
Ot framework was the critical toolkit that enabled cross- 
platform support. Little work has to be done to deliver a 
version of Viz on a wide range of platforms. Although the 
primary MER delivery platform is Linux, we have at 
various times released Windows, Irix, and Mac OS X 
versions of Viz. 




The base functionality of Viz can be divided into four 
components: 1) binary plug-in component coordination, 2) 
inter-application communication, 3) internal 3D database 
management, and 4) user interface. Figure 2 is a schematic 
representation of this architecture. The signal-slot 
mechanism of the Qt framework is utilized to provide a 
message-passing infrastructure to connect binary plug-in 
components in a publish-subscribe model. Each binary 
plug-in component is a dynamically linked library and can 
initiate its own execution thread. The four base 
components are binary plug-in components themselves, 
and a discussion of each of the base components follows. 



Figure 2. The Viz architecture. The system’s functionality 
is divided into four base components. 

2.1.1 Plug-in component coordination 

The coordination between all the binary plug-in 
components occurs in the “Core” component. This 
component determines the run-time configuration (i.e., 
which components to load), loads all other components, 
and establishes the message-passing links. 

Since components are self-contained and communicate 
through a common message infrastructure managed by the 
Core component, additional functionality for specific 
application specific tasks can be implemented in separate 
components and selectively loaded at runtime. These 
additional plug-in components operate transparently with 
the base system providing an extensible integrated feature 
set. 


2.1.2 Inter-application Communication 

Initially the primary means of inter-component 
coordination and communication, the inter-application 
communication component is now simply another binary 
plug-in component that acts as an uiterniediary between the 
Core component and external network components. 

This component implements a socket-based 
communications layer using the External Data 
Representation Standard (XDR) protocol. XDR is an open 
standard [9] first used in the MPL version of Viz. 
Messages are asynchronously converted between the 
internal Viz format and the XDR protocol, enabling 
external processes, whether local or remote, to function for 
most purposes as if they were binary plug-in components. 


2.1.3 Internal 3D Database Management 

The Openlnventor high-level graphics library is used 
for rendering, modifying, and constructing the internal 3D 
scene graph database in the 3D database management 
component. Openlnventor was also used in the MPL 
version of Viz, and the functionality implemented in this 
component along with the inter-application communication 
component and the UI component formed the conceptual 
basis for the original “visualization server”. 

This component supports loading 3D model data files 
conforming to the following specifications into the 3D 
environment: 

• JPL’s initial VISTA specification 

3 JPL’s initial ASD specification 

• Virtual Reality Modeling Language (VRML) version 

1.0 

• Openlnventor version 2. 1 file format 

• JPL’s ITS A file format used with WITS for FIDO 

Input files in the VRML and Openlnventor formats 
may utilize auxiliary image files that define 3D model 
textures. Image files in JPEG, RGB, GIF, and Planetary 
Data System (PDS) formats are supported. In addition to 
3D file formats and image formats, Viz supports import of 
target files conforming to the ARES RML uplink target 
file format. 

Manipulation and interrogation of the internal 3D 
database is handled through a queue of requests from other 
components. For complex environments, the time to 
process a request may be large, so the queue is optimized 
to improve efficiency by compressing multiple requests 
into one action. For example, during animated sequences, 
a scene element such as the rover camera mast, may be 
requested to rotate in small increments. If messages arrive 
faster than they can be processed, they are automatically 
composed into a single request representing the net effect 
of the rotation. 


2.1.4 User Interface 

The Qt graphical user interface classes are utilized to 
automatically provide a native look and feel for each 
supported platform. Two custom window classes are 
employed to logically divide functionality: 1) user initiated 
management of the internal 3D database is accomplished 
through a dedicated "main" window, and 2) multiple, 
interactive renderings of the environment are presented 
through "viewer" windows. In each window class, a 
docking toolbar paradigm allows access to functionality, 
with dialog windows for detailed tasks. As a user interacts 
with the simulated environment, action requests are sent to 
the internal 3D database management component for 


execution. The response is received and displayed by the 
user interface component. 


2.1.5 Application Specific Components 

For the MER mission, in addition to the base 
components, four application-specific binary plug-ins were 
developed: 1) A kinematic simulation component based on 
the VirtualRobot program developed at the Ecole 
Polytechnique Federale Lausanne (EPFL) and NASA ARC 
[10]. This component provides a forward and inverse 
kinematics simulation capability, 2) A solar system 
ephemeris component. This is utilized for time of day 
lighting simulation and planetary body location 
functionality, and utilizes the JPL SPICE libraries, 3) a 
CEP component allowing access to data in the CIP 
database, and 4) a MERBoard component implementing 
the MERBoard client interface. 

A network component was also developed that could 
parse MER engineering telemetry files, allowing playback 
of executed rover command sequences in the Viz 
environment. 

3 Surface Reconstruction 

The Ames Stereo-pipeline incorporates a fast image 
correlation algorithm for calculating positional disparities 
between corresponding points in left and right images of a 
stereo pair. From the image disparities, 3D points are 
calculated using the known camera geometry, and these 
points are used to create a 3D triangle mesh. In addition, 
various pre- and post-processing modules in the pipeline 
can be invoked to condition the data for improved 
performance when correlation is difficult, or to optimize 
the generated mesh. 

As with MPL, image rectification and distortion 
removal was primarily achieved with JPL’s GDS image 
processing tools. However, it was still necessary to adapt 
Ames’ stereo pipeline to accept the PDS image file format 
used by the mission, and to read camera model information 
from image headers. For MER, the camera geometry and 
lens distortion information is specified using variants of 
the “CAHV” model [11]. 

The Cornell Pancam team provided accurate 
radiometric calibration information for Pancam images in 
derived image product headers. The Ames Stereo-pipeline 
was adapted to use this information to correct for image 
brightness differences due to varying lighting conditions 
over the duration of an image panorama acquisition. The 
Pancam team also provided initial color adjustment 
information that allowed us to generate near true color 3D 
model textures from sets of monochrome images acquired 
with the Pancams color filters. 


3.1 Non-standard 3D Data Products 

In addition to standard 3D data products generated 
from matched left-right stereo image pairs, 3D models were 
also generated from individual overlapping • images. In 
particular, the Pancams have a somewhat narrow field of 
view, which prevents them from imaging targets close to 
the rover in left and right cameras simultaneously. 
However, scientists attempted to overcome this limitation 
by moving the Pancam Mast Assembly (PMA) between 
two individual image acquisitions to achieve overlap. To 
build 3-D models from these somewhat unorthodox stereo 
pairs, the assumption was that the PMA kinematics were 
well enough understood that camera position telemetry was 
essentially exact. In practice, the only artifact evident was a 
minimal amount of warping near the edges. The scale of 
objects reconstructed in this way is also not as precise as 
with calibrated Pancam stereo cameras pairs. The same 
technique was used to reconstruct 3-D models from pairs of 
microscopic images, where camera repositioning is 
achieved with motion of the IDD. Finally, 3D models 
from overlapping Mars Global Surveyor (MGS) and MER 
descent imagery were generated. Scale was roughly 
calculated from knowledge of the orbiters and altitude and 
a planarity assumption 

4 MER Science Support Experiences 

Members of the Viz, MERBoard, CIP, and MI 
Toolkit teams were on-site at JPL during the first 90 days 
of the MER mission. The Viz and MI Toolkit teams in 
particular worked closely together due to commonalities in 
data processing tools and the complimentary nature of the 
data products generated by the two teams. Our presence at 
JPL proved to be critical to our successful support of the 
MER science team. In the following we describe the joint 
support provided by the Viz and MI Toolkit teams, and 
lessons learned from the experience. 

4.1 Science Support Tasks 

Technologists’ naturally want to tune any tools they 
develop to address the most typical challenges with which 
they are confronted. For efficient production of good 
quality visualizations, the obvious approach was to ensure 
that the stereo reconstruction processes we used behaved 
well on the most common sorts of terrain. Unfortunately, 
the scientists’ interests often were in direct conflict with 
this strategy. Their attention focused most immediately on 
the closest atypical terrain features, as these are the things 
that were the most interesting to them. As a result, one of 
the greatest assets we had as a team was the ability to adapt 
our tools and methods “on the fly” to meet the varied 
needs of the science team. 


4.1.1 Microscopic Imager Data Analysis 

One of our more successful efforts during the mission 
was to support the analysis of cross bedding (layering) in 
the rocks at the Opportunity landing site. Images of the 



rocks taken with the Pancam sometimes did not provide 
sufficient resolution for in depth science analysis. The 
microscopic imager (MI) offered much better resolution of 
the rock surface. However, the MI has an extremely narrow 
field of view (only about 2mm by 5mm), and has only a 
single camera, which makes stereo reconstruction 
problematic. 

Our team worked with the mission scientists to create 
and register a mosaic of approximately 120 MI images and 
we adapted the Ames Stereo-pipeline to reconstruct 3-D 
models of the rock’s surface based on knowledge of the 
movement of the rover’s Instrument Deployment Device 
(IDD) between successive MI images. The models 
constructed using these opportunistically developed 
processes allowed the mission scientists to take 
measurements of the geometry of the rocks at the landing 
site to support their analysis of cross-bedding, 

Another analysis task we performed while on site at 
JPL, was the creation of models where MI and Pancam 
stereo data were “fused” and geometrically registered, so 
that scientists were able to see (for example) the overall 
shape of a rock, with a highly detailed patch of MI data 
correctly registered at an area of interest (see Figure 3). As 
with the MI mosaic described above, we were able to adapt 
our tools and methods to an unanticipated requirement 
rapidly enough to provide additional analytical support to 
the scientists during the mission. 


4.1.2 Terrain Visualization Support 

The creation of accurate, smooth terrain models from 
stereo imagery is part art and part science. While the JPL 
GDS image processing tools did construct stereo models 
from a portion of the stereo image data available, they were 
not optimized for visual quality or’ to support specific 
science analysis - the data processing volume was simply 
too high, and rover operational concerns were paramount. 
Since our team was focused solely on science activities and 
analysis, we were able to devote our time to working with 
the scientists to identify specific areas of interest and then 
optimize the 3-D models of those areas. 

As described above, Viz provides for the viewing of 
3-D terrain models in stereo with hardware support for 
LCD shutter glasses. This proved to be a valuable tool for 
the science team to gain contextual awareness of the rover’s 
surroundings, as terrain features that are not obvious in 2-D 
projections are sometimes very prominent when viewed in 
3-D. The geology and long term planning theme groups of 
the science team in particular made frequent use of stereo 
viewing capabilities. 



Figure 3. A 3-D model of the rock “King 3” at the MER-B 
landing site. The false color model was created from 
Pancam imagery, with a 3-D MI model superimposed over 
an area that was surfaced with the RAT. 

4.2 Data Browsing/Management 

“Round-trip data tracking” was not fully supported by 
the official mission software tools, because of limited time 
and resources prior to the landing of the rovers. Because of 
the complex workflow that an activity plan followed before 
being delivered to the rovers, and the high volume of data 
returned from the rovers, it was a challenge to correlate the 
returned data with the plan that had initiated its collection. 
Absent conferral with the science team member or group 
that had created a particular plan, it was very difficult to 
“reverse-engineer” the purpose behind a given collection of 
returned images. 

We developed two interfaces to help organize data 
access. The “Viz Browser” cataloged the 3-D models our 
team produced, and connected them to the original 2-D 
images used to generate the 3-D models. Similarly, the 
“Microscopic Image Browser” helped to organize the data 
returned from the MI and correlate raw images with derived 
data products. Both interfaces were accessed through a 
common “Ames Browser” web interface. 

4.3 V a!ue of Direct Contact with Science Team 

Ultimately, there were two major “lessons learned” 
from our MER experiences, both of which argue strongly 
for the value of having technical teams present during the 
mission. First, the tight schedules and odd hours 
(synchronized to Martian rather than Earth days) made 
external collaboration all but impossible. It would have 
been very difficult to coordinate with members of the 
science team if we had not been there working under the 
same schedule and deadline pressures. Second, the extreme 
scheduling demands of mission operations produced a very 
reasonable “expedience driven” mindset amongst the 
scientists. Whenever possible, the scientists preferred to 
use their own computers and familiar tools such as 
Photoshop and PowerPoint. The scientists overriding 
priority was to get their tasks completed in the fastest most 
direct way possible. What this meant for us was that the 



real resource we provided was our presence at JPL and our 
willingness and ability to adapt our processes and tools to 
support a variety of science analysis tasks, not the software 
tools we developed in and of themselves. This was true for 
the analytical tools and data products we provided, and 
even to a certain degree for mission critical GDS tools. 

The demanding schedule and huge volume of tasks 
required for mission operations means that it is probably 
not realistic to expect scientists to develop expertise in all 
of the software tools available on a typical mission, due to 
the wide variety. Conversely, having a technical team 
available and ready to support a variety of science tasks 
with a set of tools familiar to them, proved to be very 
productive and valuable. 

5 Conclusions 

As both a standalone application and networked 
graphics server, Viz delivers a high degree of flexibility to 
support the needs of NASA's evolving robotic planetary' 
exploration program. During the MER mission, Viz and 
the Ames Stereo-pipeline were able to be adapted by the 
on-site support team to provide scientists with valuable 
information that was not anticipated before the landed 
operations. With the trend of increasing complexity and 
higher expectations for NASA planetary exploration 
missions, the need for unanticipated analysis capabilities is 
likely to continue, and even increase. The experiences of 
this team suggest that, in order to maximize the return 
from these missions, it is critical to incorporate and utilize 
system elements that can quickly and easily change to meet 
unforeseen needs. 
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